Image processing device, image processing method, and program

ABSTRACT

An image processing device includes a normal light image acquisition section that acquires a normal light image that includes an object image and includes information within a wavelength band of white light, a special light image acquisition section that acquires a special light image that includes an object image and includes information within a specific wavelength band, a correction section that performs a correction process on the special light image, and a blending section that performs a blending process that blends the normal light image and a corrected special light image that is the special light image corrected by the correction section. The blending section blends a component G of the normal light image and a component G of the corrected special light image, and blends a component B of the normal light image and a component B of the corrected special light image.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. Ser. No.13/069,817, filed Mar. 23, 2011, which claims the benefit of priority ofJapanese Patent Application No. 2010-65925 filed on Mar. 23, 2010, eachof which is hereby incorporated by reference in its entirety.

BACKGROUND

The present invention relates to an image processing device, an imageprocessing method, a program, and the like.

A frame sequential endoscope system that sequentially applies threecolors of light (R1, G1, and B1) to tissues in a body cavity using arotary filter, and allows diagnosis using an image (normal light image)generated from the resulting reflected light images, has been widelyused. JP-A-2006-68113 discloses an endoscope system that sequentiallyapplies narrow-band light G2 and narrow-band light B2 that differ fromthe above three colors of light to tissues in a body cavity, and allowsdiagnosis using a narrow-band light image (special light image)generated from the resulting reflected light images.

When using an endoscope system that acquires a special light image(e.g., JP-A-2006-68113), capillaries and a minute mucous membranepattern in the mucous membrane surface layer are enhanced (highlighted),so that a lesion area (e.g., epidermoid cancer) that cannot be easilyobserved using normal light can be easily found.

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing device comprising:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band;

a correction section that performs a correction process on the speciallight image; and

a blending section that performs a blending process that blends thenormal light image and a corrected special light image that is thespecial light image corrected by the correction section,

the blending section performing at least one of a first blending processthat blends a component G of the normal light image and a component G ofthe corrected special light image, and a second blending process thatblends a component B of the normal light image and a component B of thecorrected special light image as the blending process.

According to another aspect of the invention, there is provided an imageprocessing device comprising:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band; and

a correction section that performs a correction process on the speciallight image based on special light luminance information that isluminance information about the special light image.

According to another aspect of the invention, there is provided an imageprocessing method comprising:

acquiring a normal light image that includes an object image andincludes information within a wavelength band of white light;

acquiring a special light image that includes an object image andincludes information within a specific wavelength band;

performing a correction process on the special light image; and

performing at least one of a first blending process that blends acomponent G of the normal light image and a component G of the correctedspecial light image, and a second blending process that blends acomponent B of the normal light image and a component B of the correctedspecial light image as a blending process that blends the normal lightimage and the corrected special light image.

According to another aspect of the invention, there is provided aprogram stored in an information storage medium, the program causing acomputer to function as:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band;

a correction section that performs a correction process on the speciallight image; and

a blending section that performs a blending process that blends thenormal light image and a corrected special light image that is thespecial light image corrected by the correction section,

the blending section performing at least one of a first blending processthat blends a component G of the normal light image and a component G ofthe corrected special light image, and a second blending process thatblends a component B of the normal light image and a component B of thecorrected special light image as the blending process.

According to another aspect of the invention, there is provided aprogram stored in an information storage medium, the program causing acomputer to function as:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band; and

a correction section that performs a correction process on the speciallight image based on special light luminance information that isluminance information about the special light image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are views illustrative of a related-art method.

FIG. 2 is a view illustrative of a method according to one embodiment ofthe invention.

FIG. 3 is a view illustrative of a component G blending processaccording to one embodiment of the invention.

FIG. 4 is a view illustrative of a component B blending processaccording to one embodiment of the invention.

FIG. 5 shows a system configuration example according to one embodimentof the invention.

FIG. 6 shows the spectral characteristics of color filters r, g, and b.

FIG. 7 shows the spectral characteristics of color filters g2 and b2.

FIG. 8 is a view illustrative of color filters g2 and b2.

FIG. 9 shows a configuration example of a normal light image acquisitionsection.

FIG. 10 shows a configuration example of a special light imageacquisition section.

FIG. 11 shows a configuration example of an output image generationsection.

FIG. 12 shows a configuration example of a blending section.

FIG. 13 shows another configuration example of a blending section.

FIGS. 14A to 14D are views illustrative of a pixel and a direction usedwhen determining the edge direction.

FIG. 15 shows a configuration example of a computer used for a softwareprocess.

FIG. 16 shows a configuration example of a computer used for a softwareprocess.

FIG. 17 is a flowchart illustrative of a process according to oneembodiment of the invention.

FIG. 18 is a flowchart illustrative of a blending process.

FIG. 19 is another view illustrative of a component G blending processaccording to one embodiment of the invention.

FIG. 20 is another view illustrative of a component B blending processaccording to one embodiment of the invention.

FIG. 21 shows another configuration example of a blending section.

FIG. 22 is a flowchart illustrative of a blending process.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Several aspects of the invention may provide an image processing device,an image processing method, a program, and the like that improve thevisibility of an object (e.g., blood vessel) as compared with the caseof blending a pseudo-color special light image, by acquiring a normallight image corresponding to the wavelength region of normal light and aspecial light image corresponding to a specific wavelength region,correcting the special light image, and blending the components G andthe components B of the corrected special light image and the normallight image.

Several aspects of the invention may provide an image processing device,an image processing method, a program, and the like that prevent asituation in which a lesion area is missed while reducing the burden onthe doctor during diagnosis that utilizes a normal light image and aspecial light image.

According to one embodiment of the invention, there is provided an imageprocessing device comprising:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band;

a correction section that performs a correction process on the speciallight image; and

a blending section that performs a blending process that blends thenormal light image and a corrected special light image that is thespecial light image corrected by the correction section,

the blending section performing at least one of a first blending processthat blends a component G of the normal light image and a component G ofthe corrected special light image, and a second blending process thatblends a component B of the normal light image and a component B of thecorrected special light image as the blending process.

According to the above embodiment, the normal light image and thespecial light image are acquired. The special light image is correctedto acquire a corrected special light image, and the components G and thecomponents B of the corrected special light image and the normal lightimage are blended. Therefore, the visibility of the object (e.g., bloodvessel) can be improved as compared with the case of blending apseudo-color special light image with the normal light image.

According to another embodiment of the invention, there is provided animage processing device comprising:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band; and

a correction section that performs a correction process on the speciallight image based on special light luminance information that isluminance information about the special light image.

According to the above embodiment, the normal light image and thespecial light image are acquired, and the correction process isperformed on the special light image based on the brightness informationabout the special light image. This makes it possible to obtain a brightspecial light image as compared with the case where the correctionprocess is not performed, so that the visibility of the image isimproved.

According to another embodiment of the invention, there is provided animage processing method comprising:

acquiring a normal light image that includes an object image andincludes information within a wavelength band of white light;

acquiring a special light image that includes an object image andincludes information within a specific wavelength band;

performing a correction process on the special light image; and

performing at least one of a first blending process that blends acomponent G of the normal light image and a component G of the correctedspecial light image, and a second blending process that blends acomponent B of the normal light image and a component B of the correctedspecial light image as a blending process that blends the normal lightimage and the corrected special light image.

According to another embodiment of the invention, there is provided aprogram stored in an information storage medium, the program causing acomputer to function as:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band;

a correction section that performs a correction process on the speciallight image; and

a blending section that performs a blending process that blends thenormal light image and a corrected special light image that is thespecial light image corrected by the correction section,

the blending section performing at least one of a first blending processthat blends a component G of the normal light image and a component G ofthe corrected special light image, and a second blending process thatblends a component B of the normal light image and a component B of thecorrected special light image as the blending process.

According to another embodiment of the invention, there is provided aprogram stored in an information storage medium, the program causing acomputer to function as:

a normal light image acquisition section that acquires a normal lightimage that includes an object image and includes information within awavelength band of white light;

a special light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band; and

a correction section that performs a correction process on the speciallight image based on special light luminance information that isluminance information about the special light image.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note that all of the elements ofthe following exemplary embodiments should not necessarily be taken asessential elements of the invention.

1. Method

A method according to one embodiment of the invention is described belowwith reference to FIGS. 1A, 1B, and 2.

FIGS. 1A and 1B show a related-art method. FIG. 1A shows an observationstate using normal light. A bright image that can be easily observed isobtained using normal light. However, it is difficult to observe somelesion such as epidermoid cancer. FIG. 1B shows an observation stateusing special light (narrow-band light). In this case, the visibility ofsome lesion can be improved (e.g., some lesion such as epidermoid canceris displayed in brown) as compared with observation using normal light.However, a dark image that cannot be easily observed is obtained usingspecial light.

In order to solve this problem, diagnosis or treatment may be performedwhile selectively displaying the normal light image or the special lightimage by operating a switch of the instrument, for example. According tothis method, however, the burden on the doctor increases since it isnecessary to operate the instrument and observe the screen while movingthe insertion section of the endoscope. Moreover, since each of thenormal light image and the special light image has drawbacks, it isnecessary to appropriately select the display image depending on thesituation. This may require skill.

The normal light image and the special light image may be displayed sideby side. In this case, since it is necessary to observe two screens(images) at the same time, the burden on the doctor increases. As aresult, a lesion area may be missed, for example.

In order to deal with the above problems, this application proposes asystem shown in FIG. 2. Specifically, the visibility of a lesion area(e.g., epidermoid cancer) is improved by extracting capillaries and aminute mucous membrane pattern in the mucous membrane surface layer fromthe special light image, and performing the blending process on thespecial light image and the normal light image based on the extractedinformation.

Since the advantage (i.e., visibility of a lesion area) of the speciallight image is thus added to the advantage (i.e., bright and easilyobservable) of the normal light image, smooth diagnosis or treatment canbe implemented by preventing a situation in which a lesion area ismissed, and reducing the burden on the doctor.

Note that the special light image may also be displayed by operating aswitch of the instrument, for example. For example, the blended normallight image may be used when searching for a lesion area, and thespecial light image may be used when observing the details of the lesionarea (i.e., the images may be selectively used).

2. First Embodiment

A method according to this embodiment is described below with referenceto FIGS. 3 and 4.

FIG. 3 is a view illustrative of a component G blending process. Thespecial light image includes components G2 and B2 (when the speciallight image is an NBI image), and the component G2 is extracted. Thenormal light image includes components R, G, and B, and the component Gthat corresponds to the component G2 is extracted.

A ratio ratioG is calculated by the expression (3) (described later)from the ratio of luminance information wliAveG about the component G ofthe normal light image to luminance information nbiAveG about thecomponent G2 of the special light image. The luminance of the componentG2 of the special light image is corrected by the expression (4)(described later) using the ratio ratioG to acquire the correctedspecial light image (component G).

The corrected special light image and the component G of the normallight image are blended by the expression (24) (described later) using ablending ratio blendG to acquire an output image (component G). Notethat the process according to this embodiment aims at acquiring theoutput image. The blending ratio blendG is calculated the followingprocess.

When calculating the blending ratio blendG, the corrected special lightimage is subjected to a Laplacian filtering process to acquire an edgeimage lapImgG. The edge direction in the edge image lapImgG, isdetermined using the expressions (8) to (15) (described later), and animage AveG is acquired by applying a direction-dependent filteringprocess that calculates the average value of the pixel values along theedge direction.

The image AveG is subjected to a coring process (expressions (16) to(20) (described later) to acquire an edge information image edgeG. Theblending ratio blendG is acquired by multiplying the edge informationimage edgeG by a coefficient K1, and the blending process is performedon the component G of the normal light image and the corrected speciallight image using the acquired blending ratio blendG to acquire anoutput image.

Likewise, the component B2 of the special light image is blended withthe component B of the normal light image. FIG. 4 schematically shows acomponent B blending process.

The component B blending process is performed in the same manner as thecomponent G blending process. Specifically, a ratio ratioB is calculatedfrom the ratio of luminance information about the component B of thenormal light image to luminance information about the component B2 ofthe special light image, and the luminance of the component B2 of thespecial light image is corrected. The component B of the normal lightimage and the corrected special light image (component B) are blendedusing a blending ratio blendB to acquire an output image (component B).

The blending ratio blendB is calculated by subjecting the correctedspecial light image to the Laplacian filtering process and thedirection-dependent filtering process, calculating an edge informationimage edgeB by performing the coring process, and multiplying the edgeinformation image edgeB by a coefficient K2.

Since the special light image does not include a component correspondingto the channel R, the component R of the normal light image is used asthe component R of the output image (expression (23) (described later)).

A color image is generated based on the components R, B, and G of theoutput image thus obtained. Since the blood vessels and the like areenhanced in a natural color in the generated image, the visibility ofthe structure (e.g., blood vessels) and a lesion area is improved. Thismakes it possible to reduce the burden on the doctor during diagnosis,for example.

The system configuration and the above process are described in detailbelow.

An image processing device according to this embodiment and an endoscopesystem including the same are described below with reference to FIG. 5.The endoscope system according to this embodiment includes a lightsource section 100, an insertion section 200, an image processingsection 300 that corresponds to the image processing device according tothis embodiment, a display section 400, and an external I/F section 500.

The light source section 100 includes a white light source 110 thatemits (generates) white light, and a condenser lens 120 that focuseswhite light on a light guide fiber 210.

The insertion section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the insertion section 200 can be inserted into abody cavity or the like. The insertion section 200 includes the lightguide fiber 210 that guides light focused by the light source section100, an illumination lens 220 that diffuses light that has been guidedby the light guide fiber 210, and illuminates an observation target, anobjective lens 230 that focuses light reflected by the observationtarget, a half mirror 240 that separates the focused reflected lightinto two, and a first imaging element 250 and a second imaging element260 that detect the separated reflected light.

The first imaging element 250 includes a Bayer color filter array thatis used to capture the normal light image. Color filters R, G, and B ofthe first imaging element 250 have spectral characteristics shown inFIG. 6, for example. The second imaging element 260 captures anarrow-band light image. As shown in FIG. 8, the second imaging element260 has a configuration in which color filters g2 that allow narrow-bandlight G2 to pass through and color filters b2 that allow narrow-bandlight B2 to pass through are disposed in a staggered arrangement, forexample. As shown in FIG. 7, the color filter g2 of the second imagingelement 260 allows light within a wavelength band of 530 to 550 nm topass through, and the color filter b2 of the second imaging element 260allows light within a wavelength band of 390 to 445 nm to pass through,for example.

The image processing section 300 (image processing device) includes ADconversion sections 310 a and 310 b, a normal light image acquisitionsection 320, a special light image acquisition section 330, an outputimage generation section 340, a selection section 350, and a controlsection 360. The control section 360 is bidirectionally connected to thenormal light image acquisition section 320, the special light imageacquisition section 330, the output image generation section 340, andthe selection section 350, and controls the normal light imageacquisition section 320, the special light image acquisition section330, the output image generation section 340, and the selection section350.

The external I/F section 500 is an interface that allows the user toperform an input operation or the like on the image processing device(endoscope system). The external I/F section 500 includes a power supplyswitch (power supply ON/OFF switch), a shutter button (photographingstart button), a mode (e.g., photographing mode) change button, and thelike. The external I/F section 500 outputs input information to thecontrol section 360.

The AD conversion section 310 a converts an analog signal output fromthe first imaging element 250 into a digital signal, and outputs thedigital signal. The AD conversion section 310 b converts an analogsignal output from the second imaging element 260 into a digital signal,and outputs the digital signal.

The normal light image acquisition section 320 acquires the normal lightimage based on the digital signal output from the AD conversion section310 a, for example. The special light image acquisition section 330acquires the special light image based on the digital signal output fromthe AD conversion section 310 b, for example. The normal light imageacquisition section 320 and the special light image acquisition section330 are described in detail later.

The normal light image acquired by the normal light image acquisitionsection 320 and the special light image acquired by the special lightimage acquisition section 330 are output to the output image generationsection 340. The special light color image acquired by the special lightimage acquisition section 330 is output to the selection section 350.The output image generation section 340 generates one output image fromthe normal light image and the special light image, and outputs theoutput image to the selection section 350. The selection section 350outputs a display image to the display section 400. The output imagegeneration section 340 is described in detail later.

The normal light image acquisition section 320 is described below withreference to FIG. 9. The normal light image acquisition section 320includes a normal light image generation section 321, and a normal lightimage storage section 322. The AD conversion section 310 a is connectedto the normal light image generation section 321. The normal light imagegeneration section 321 is connected to the normal light image storagesection 322. The normal light image storage section 322 is connected tothe output image generation section 340.

When a digital signal converted by the AD conversion section 310 a hasbeen input to the normal light image generation section 321, the normallight image generation section 321 processes the digital signal togenerate a normal light image. Specifically, the normal light imagegeneration section 321 performs an interpolation process, a whitebalance process, a color conversion process, a grayscale transformationprocess, and the like on the digital signal to generate a normal lightimage. The R, G, and B signal values at the coordinates (x, y) of thenormal light image are indicated by r(x, y), g(x, y), and b(x, y),respectively. The normal light image storage section 322 stores thenormal light image output from the normal light image generation section321. The normal light image stored in the normal light image storagesection 322 is output to the output image generation section 340.

The special light image acquisition section 330 is described below withreference to FIG. 10. The special light image acquisition section 330includes a special light image generation section 331, and a speciallight image storage section 332. The AD conversion section 310 b isconnected to the special light image generation section 331. The speciallight image generation section 331 is connected to the special lightimage storage section 332 and the selection section 350. The speciallight image storage section 332 is connected to the output imagegeneration section 340.

When a digital signal converted by the AD conversion section 310 b hasbeen input to the special light image generation section 331, thespecial light image generation section 331 processes the digital signalto generate a special light image.

The special light image generation section 331 generates the speciallight image as follows. A digital image signal input to the speciallight image generation section has a configuration in which the colorfilters g2 and b2 are disposed in a staggered arrangement (see FIG. 8).The special light image generation section 331 performs an interpolationprocess on the image signal to generate a G2 image in which all of thepixels have a signal value of the filter g2, and a B2 image in which allof the pixels have a signal value of the filter b2. The G and B signalvalues at the coordinates (x, y) of the special light image areindicated by g2(x, y) and b2(x, y), respectively. The pixel valuecalculated by the interpolation process may be the average value of thefour peripheral pixels, for example. For example, the pixel valueb2(1,1) at the position g2(1,1) and the pixel value g2(1,2) at theposition b2(1,2) shown in FIG. 8 are calculated by the followingexpressions (1) and (2).b2(1,1)=[b2(0,1)+b2(1,0)+b2(1,2)+b2(2,1)]/4  (1)g2(1,2)=[g2(0,2)+g2(1,1)+g2(1,3)+g2(2,2)]/4  (2)

The special light image storage section 332 stores the special lightimage output from the special light image generation section 331. Thespecial light image stored in the special light image storage section332 is output to the output image generation section 340.

A specific configuration of the output image generation section 340 isdescribed below. FIG. 11 is a block diagram illustrative of an exampleof the configuration of the output image generation section 340according to the first embodiment. The output image generation section340 includes a correction section 341 and a blending section 342.

The image signal from the normal light image acquisition section 320 isoutput to the correction section 341 and the blending section 342. Theimage signal from the special light image acquisition section 330 isoutput to the correction section 341. The correction section 341 isconnected to the blending section 342. The blending section 342 isconnected to the selection section 350. The blending section 342 isdescribed in detail later. The control section 360 is bidirectionallyconnected to the correction section 341 and the blending section 342,and controls the correction section 341 and the blending section 342.

The correction section 341 performs a correction process on the speciallight image under control of the control section 360 based on the ratioof the feature quantity of the pixels in the normal light image to thefeature quantity of the pixels in the special light image. The term“correction process” used herein refers to a process that corrects thespecial light image to have a luminance almost equal to that of thenormal light image. In this embodiment, the signal G in each image isused as the feature quantity. The average value of the signals G in thespecial light image is indicated by nbiAveG, and the average value ofthe signals G in the normal light image is indicated by wliAveG. Theratio ratioG of the signal G of the normal light image to the signal Gof the special light image is calculated as follows.ratioG=K0×wliAveG/nbiAveG  (3)

Note that K0 is a constant term. The signals G and the signals B of allof the pixels of the special light image are multiplied by the signalratio ratioG. The G and B signals at the coordinates (x, y) of thespecial light image multiplied by the signal ratio (corrected speciallight image) are indicated by g2′(x, y) and b2′(x, y), respectively.g2′(x, y)=ratioG×g2(x, y)  (4)b2′(x, y)=ratioG×b2(x, y)  (5)

The signal ratio may be calculated by a method other than the abovemethod. For example, the signal ratio ratioB may be calculated using theaverage value of the signals B in the special light image and theaverage value of the signals B in the normal light image, and the signalB in the special light image may be multiplied by the signal ratioratioB.ratioB=K0×wliAveB/nbiAveB  (6)b2′(x, y)=ratioB×b2(x, y)  (7)

The image may be divided into a plurality of N×M pixel areas, theaverage value of each area may be calculated, and the signal ratio ofeach area may be calculated, for example. The image may be divided intoa plurality of areas by applying a known area division algorithm (e.g.,texture analysis). The corrected special light image is output to theblending section 342.

The blending section 342 blends the corrected special light image withthe normal light image under control of the control section 360. Aspecific configuration of the blending section 342 is described below.FIG. 12 is a block diagram illustrative of an example of theconfiguration of the blending section 342 according to the firstembodiment. As shown in FIG. 12, the blending section 342 includes anedge extraction section 3421, a blending ratio calculation section 3422,and a blended image generation section 3423.

The normal light image acquisition section 320 is connected to the edgeextraction section 3421 and the blended image generation section 3423.The correction section 341 is connected to the edge extraction section3421. The edge extraction section 3421 is connected to the blendingratio calculation section 3422. The blending ratio calculation section3422 is connected to the blended image generation section 3423. Theblended image generation section 3423 is connected to the selectionsection 350. The control section 360 is bidirectionally connected to theedge extraction section 3421, the blending ratio calculation section3422, and the blended image generation section 3423, and controls theedge extraction section 3421, the blending ratio calculation section3422, and the blended image generation section 3423.

The edge extraction section 3421 extracts edge information about thecorrected special light image and edge information about the normallight image under control of the control section 360. The edgeextraction section 3421 performs a direction determination edgeextraction process on the G signal value and the B signal value of thespecial light image.

An example of the direction determination edge extraction process on theG signal value of the special light image is described below. Thedirection determination edge extraction process is similarly performedon the B signal value of the special light image. Specifically, thesignal G of the special light image is subjected to the Laplacianfiltering process to generate the edge image lapImgG.

A direction in which an edge component is included is determined fromeight directions using all of the pixels of the edge image lapImgG.FIGS. 14A to 14D show a pixel and a direction used when determining theedge direction.

Specifically, an evaluation value is calculated using peripheral 5×5pixels of the attention pixel, and a direction having the minimumevaluation value is used as the determination result. The followingexpressions (8) to (15) are evaluation expressions when the coordinatesof the attention pixel are (2, 2).e0G(2,2)=|g2′(2,2)−g2′(2,3)|+|g2′(2,2)−g2′(2,4)|+|g2′(2,2)−g2′(2,1)|+|g2′(2,2)−g2′(2,0)|  (8)e1G(2,2)=|g2′(2,2)−g2′(2,3)|+|g2′(2,2)−g2′(1,4)|+|g2′(2,2)−g2′(2,1)|+|g2′(2,2)−g2′(3,0)|  (9)e2G(2,2)=|g2′(2,2)−g2′(1,3)|+|g2′(2,2)−g2′(0,4)|+|g2′(2,2)−g2′(3,1)|+|g2′(2,2)−g2′(4,0)|  (10)e3G(2,2)=|g2′(2,2)−g2′(1,2)|+|g2′(2,2)−g2′(0,3)|+|g2′(2,2)−g2′(3,2)|+|g2′(2,2)−g2′(4,1)|  (11)e4G(2,2)=|g2′(2,2)−g2′(1,2)|+|g2′(2,2)−g2′(0,2)|+|g2′(2,2)−g2′(3,2)|+|g2′(2,2)−g2′(4,2)|  (12)e5G(2,2)=|g2′(2,2)−g2′(1,2)|+|g2′(2,2)−g2′(0,1)|+|g2′(2,2)−g2′(3,2)|+|g2′(2,2)−g2′(4,3)|  (13)e6G(2,2)=|g2′(2,2)−g2′(1,1)|+|g2′(2,2)−g2′(0,0)|+|g2′(2,2)−g2′(3,3)|+|g2′(2,2)−g2′(4,4)|  (14)e7G(2,2)=|g2′(2,2)−g2′(2,1)|+|g2′(2,2)−g2′(1,0)|+|g2′(2,2)−g2′(2,3)|+|g2′(2,2)−g2′(3,4)|  (15)

The edge image lapImgG generated by the Laplacian filtering process issubjected to a filtering process along the edge direction using thedetermined result. This reduces noise of the edge image whilemaintaining the edge. Specifically, the average value AveG(x, y) of thefive pixels of the edge image used to calculate the minimum evaluationvalue using the expressions (8) to (15) is calculated, and used as theoutput value of the direction-dependent filtering process.

The coring process is then performed on the average value AveG(x, y)calculated by the direction-dependent filtering process so that noise ofthe edge image and an artifact due to a direction determination errorare removed. The coring amount is controlled so that the coring amountis large in the flat area and is small in the edge area.

Specifically, the values e0G(x, y), e2G(x, y), e4G(x, y), and e6G(x, y)calculated by the expressions (8), (10), (12), and (14) are compared,and the maximum value evaMaxG(x, y) is used as the edge amount index.The coring amount evaCoreG(x, y) of each pixel is calculated by thefollowing expressions (16) to (18) using the edge amount indexevaMaxG(x, y), a maximum coring amount coreMaxG, a minimum coring amountcoreMinG, a slope slopeG, and an edge amount index threshold value TheG(i.e., parameters supplied in advance from the outside).

When the edge amount index evaMaxG is smaller than the threshold valueTheG:evaCoreG(x, y)=coreMaxG  (16)

When the edge amount index evaMaxG is equal to or larger than thethreshold value TheG:evaCoreG(x, y)=coreMaxG−((evaMaxG(x, y)−TheG)×slopeG)  (17)

When the coring amount evaCoreG is smaller than the minimum coringamount coreMinG:evaCoreG(x, y)=coreMinG  (18)

The coring process is then performed on the output value AveG(x, y) ofthe direction-dependent filtering process by the following expressions(19) and (20) using the calculated coring amount evaCoreG(x, y) tocalculate the edge information edgeG(x, y).

When the output value AveG (x, y) is equal to or larger than 0:edgeG(x, y)=AveG(x, y)−evaCoreG(x, y)  (19)

When the edge information edgeG(x, y) has become smaller than 0, 0 issubstituted for the edge information edgeG(x, y).

When the output value AveG (x, y) is smaller than 0:edgeG(x, y)=AveG(x, y)+evaCoreG(x, y)  (20)

When the edge information edgeG(x, y) has become larger than 0, 0 issubstituted for the edge information edgeG(x, y).

Note that a known edge extraction method may be used instead of usingthe direction determination filtering process as the edge information.The edge image edgeG of the G signal value of the special light imageand the edge image edgeB of the B signal value of the special lightimage thus generated are output to the blending ratio calculationsection 3422.

The blending ratio calculation section 3422 calculates the blendingratio under control of the control section 360 based on the edgeinformation extracted by the edge extraction section 3421. Specifically,the blending ratio calculation section 3422 calculates the blendingratios blendG and blendB using the following expressions (21) and (22).blendG(x, y)=K1×edgeG(x, y)  (21)blendB(x, y)=K2×edgeB(x, y)  (22)

Note that K1 and K2 are constant terms. When the blending ratiosblendG(x, y) and blendB(x, y) are equal to or larger than 1, 1 issubstituted for the blending ratios blendG(x, y) and blendB(x, y). Whenthe blending ratios blendG(x, y) and blendB(x, y) are equal to orsmaller than 0, 0 is substituted for the blending ratios blendG(x, y)and blendB(x, y). Note that the blending ratio may be calculated by amethod other than the above method. For example, the corrected speciallight image may be divided into a plurality of areas, the average valuesedgeAveG and edgeAveB of the edge information of each area may becalculated, and the blending ratio may be calculated using the averagevalue of the edge information. The calculated blending ratios blendG andblendB are output to the blended image generation section 3423.

The blended image generation section 3423 blends the special light imageand the normal light image based on the blending ratio calculated by theblending ratio calculation section 3422 under control of the controlsection 360. Specifically, the signals R, G, and B (R(x, y), G(x, y),and B(x, y)) of the blended image are calculated by the followingexpressions.R(x, y)=r(x, y)  (23)G(x, y)=(1−blendG(x, y))×g(x, y)+blendG(x, y)×g2′(x, y)  (24)B(x, y)=(1−blendB(x, y))×b(x, y)+blendB(x, y)×b2′(x, y)  (25)

The blended image is output to the selection section 350.

The selection section 350 selects the display image under control of thecontrol section 360. In this embodiment, the selection section 350selects the special light color image or the blended image as thedisplay image based on the photographing mode selected using theexternal I/F section 500. The selected display image is output to thedisplay section 400.

Note that the configuration of the blending section 342 is not limitedto the above configuration. For example, the blending section 342 mayinclude an attention area detection section 3425, the blending ratiocalculation section 3422, and the blended image generation section 3423,as shown in FIG. 13.

The attention area detection section 3425 detects an attention area. Theattention area is not limited to the edge information (e.g., bloodvessel), but may be a lesion area or the like. The blending ratiocalculation section 3422 increases the blending ratio of the correctedspecial light image in an area where the attention area has beendetected, and decreases the blending ratio of the corrected speciallight image in an area where the attention area has not been detected.

When performing NBI observation, for example, the components G2 and B2are input to given channels among the channels R, G, and B to generate apseudo-color image, so that a specific lesion (e.g., epidermoid cancer)is displayed in brown. Therefore, a brown area (e.g., an area having ahue H of 5 to 35) can be recognized as a lesion area, and detected asthe attention area.

Note that the blending method is performed using the expressions (23) to(25). Specifically, the normal light image and the pseudo-color speciallight image (having the components R, G, and B) are not blended, but thecomponent G of the normal light image and the component G2 of thespecial light image, and the component B of the normal light image andthe component B2 of the special light image, are blended.

The reliability of the attention area may be determined, and theblending ratio of the corrected special light image may be increased asthe reliability increases. The area of the brown area (lesion area) maybe used as the reliability, for example.

In this embodiment, each section of the image processing section 300 isimplemented by hardware. Note that the configuration of the imageprocessing section 300 is not limited thereto. For example, a CPU mayperform the process of each section on an image acquired using animaging apparatus such as a capsule endoscope. Specifically, the processof each section may be implemented by software by causing the CPU toexecute a program. Alternatively, part of the process of each sectionmay be implemented by means of software.

When separately providing the imaging section and the AD conversionsection, and implementing the process of each section of the imageprocessing section 300 other than the AD conversion section by means ofsoftware, a known computer system (e.g., work station or personalcomputer) may be used as the image processing device. A program (imageprocessing program) that implements the process of each section of theimage processing section 300 may be provided in advance, and executed bythe CPU of the computer system.

FIG. 15 is a system configuration diagram showing the configuration of acomputer system 600 according to this modification. FIG. 16 is a blockdiagram showing the configuration of a main body 610 of the computersystem 600. As shown in FIG. 15, the computer system 600 includes themain body 610, a display 620 that displays information (e.g., image) ona display screen 621 based on instructions from the main body 610, akeyboard 630 that allows the user to input information to the computersystem 600, and a mouse 640 that allows the user to designate anarbitrary position on the display screen 621 of the display 620.

As shown in FIG. 16, the main body 610 of the computer system 600includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, aCD-ROM drive 615 that receives a CD-ROM 660, a USB port 616 to which aUSB memory 670 is removably connected, an I/O interface 617 thatconnects the display 620, the keyboard 630, and the mouse 640, and a LANinterface 618 that is used to connect to a local area network or a widearea network (LAN/WAN) N1.

The computer system 600 is connected to a modem 650 that is used toconnect to a public line N3 (e.g., Internet). The computer system 600 isalso connected to personal computer (PC) 681 (i.e., another computersystem), a server 682, a printer 683, and the like via the LAN interface618 and the local area network or the large area network N1.

The computer system 600 implements the functions of the image processingdevice by reading an image processing program (e.g., an image processingprogram that implements a process described later referring to FIGS. 17and 18) recorded on a given recording medium, and executing the imageprocessing program. The given recording medium may be an arbitraryrecording medium that records the image processing program that can beread by the computer system 600, such as the CD-ROM 660, the USB memory670, a portable physical medium (e.g., MO disk, DVD disk, flexible disk(FD), magnetooptical disk, or IC card), a stationary physical medium(e.g., HDD 614, RAM 612, or ROM 613) that is provided inside or outsidethe computer system 600, or a communication medium that temporarilystores a program during transmission (e.g., the public line N3 connectedvia the modem 650, or the local area network or the wide area network N1to which the computer system (PC) 681 or the server 682 is connected).

Specifically, the image processing program is recorded on a recordingmedium (e.g., portable physical medium, stationary physical medium, orcommunication medium) so that the image processing program can be readby a computer. The computer system 600 implements the functions of theimage processing device by reading the image processing program fromsuch a recording medium, and executing the image processing program.Note that the image processing program need not necessarily be executedby the computer system 600. The invention may be similarly applied tothe case where the computer system (PC) 681 or the server 682 executesthe image processing program, or the computer system (PC) 681 and theserver 682 execute the image processing program in cooperation.

A process performed when implementing the process of the output imagegeneration section 340 shown in FIG. 11 by software on the normal lightimage and the special light image acquired in advance is described belowusing a flowchart shown in FIG. 17 as an example of implementing part ofthe process of each section by software.

Specifically, header information (e.g., photographing mode) is input tothe time-series normal light image and the special light image (S11).The special light image and the normal light image are input to an imagebuffer allocated in advance (S12). An interpolation process, a whitebalance process, a color conversion process, a grayscale transformationprocess, and the like are performed on the input normal light image togenerate a normal light image (S13).

The input special light image is processed using the expressions (1) and(2) to generate a special light image (S14). The correction process isperformed on the special light image based on the ratio of the featurequantity of the pixels in the normal light image to the feature quantityof the pixels in the special light image (see the expressions (3) to(5)) (S15). The corrected special light image is then blended with thenormal light image (described in detail later with reference to FIG. 18)(S16). The image signal subjected to the blending process is then output(S17). Whether or not the process has been performed on the finaltime-series image is then determined (S18). When it has been determinedthat the process has not been performed on the final image, the aboveprocess is performed on the next image signal from the step S12. When ithas been determined that the process has been performed on all of theimage signals, the process is terminated.

The details of the blending process performed in the step S16 in FIG. 17are described below with reference to FIG. 18. Specifically, the edgeinformation about the corrected special light image is extracted usingthe expressions (8) to (20) (S21). The blending ratio is calculatedbased on the extracted edge information using the expressions (21) and(22) (S22). A blended image is then generated by blending the signals Band G of the normal light image with the signals B and G of the speciallight image using the expressions (23) to (25) (S23).

The above process makes it possible to display the normal light imageand the special light image as a single image. Therefore, it is possibleto provide an image processing device (endoscope system) and the likethat prevent a situation in which a lesion area is missed while reducingthe burden on the doctor.

According to this embodiment, the normal light image acquisition section320 shown in FIG. 5 acquires an image that includes an object image andincludes information within the wavelength band of white light, and thespecial light image acquisition section 330 acquires an image thatincludes an object image and includes information within a specificwavelength band. The correction section 341 shown in FIG. 11 performsthe correction process on the special light image, and the blendingsection 342 performs the blending process on the corrected special lightimage and the normal light image (blending process). Note that theblending process refers to at least one of a first blending process thatblends the component G of the normal light image and the component G ofthe corrected special light image (C1 in FIG. 3), and a second blendingprocess that blends the component B of the normal light image and thecomponent B of the corrected special light image (D1 in FIG. 4).

The term “component” refers to data of each of the channels R, G, and B(i.e., an output when the AD conversion section has converted an analogsignal into a digital signal).

An image in which the visibility of capillaries and a minute mucousmembrane pattern in the mucous membrane surface layer is higher thanthat of the normal light image can be obtained by blending the speciallight image corrected by the correction section 341 with the normallight image. Since the components G and the components B arerespectively blended (see C1 in FIG. 3 and D1 in FIG. 4), the visibilityof the blood vessels and the like can be improved with a natural color.Therefore, when observing tissues using an endoscope by utilizing normallight and special light, for example, it is possible to allow a lesionarea that cannot be easily detected using normal light to be accuratelydetected using special light, and display an image including the lesionarea so that the visibility is higher than that in the case where themethod according to this embodiment is not used. Therefore, the burdenon the doctor can be reduced while preventing a situation in which thelesion area is missed.

An example in which the components G and the components B arerespectively blended has been described above. Note that anotherconfiguration may be employed. Specifically, only the components G maybe blended without blending the components B, or only the components Bmay be blended without blending the components G.

As shown in FIG. 12, the blending section 342 may include the edgeextraction section 3421 and the blending ratio calculation section 3422.The edge extraction section 3421 may extract at least one of the edgeinformation about the corrected special light image and the edgeinformation about the normal light image, and the blending ratiocalculation section 3422 may calculate the blending ratio based on theextracted edge information. The blending section 342 may perform theblending process based on the calculated blending ratio.

The edge information is the edge information image edgeG shown in FIG.3, and the blending ratio is the blending ratio blendG calculated bymultiplying the edge information image edgeG by the coefficient K1 (seethe expression (21)). As shown in FIG. 3, the edge information imageedgeG is obtained by subjecting the corrected special light image to theLaplacian filtering process, the direction-dependent filtering process,and the coring process.

The edge information is an index that indicates capillaries and a minutemucous membrane pattern in the mucous membrane surface layer included inthe special light image.

Specifically, the visibility of the lesion area can be improved ascompared with the case of using only normal light by improving thevisibility of the lesion area with regard to the pixels of the speciallight image that include capillaries and a minute mucous membranepattern in the mucous membrane surface layer, and displaying the pixelsthat do not include capillaries and a minute mucous membrane pattern inthe mucous membrane surface layer using the normal light image. Thismakes it possible to reduce the burden on the doctor while preventing asituation in which the lesion area is missed.

The blending ratio calculation section 3422 may increase the blendingratio of the corrected special light image when the amount of edgesincluded in the corrected special light image is large as compared withthe blending ratio of the corrected special light image when the amountof edges included in the corrected special light image is small.Specifically, the blending ratio of the corrected special light image isincreased as the value edgeG increases. This is clear from theexpressions (21) and (24). Specifically, the blending ratio blendG ofthe corrected special light image (see the expression (24)) isdetermined by the value edgeG (see the expression (21)).

This makes it possible to improve the visibility of the lesion area byincreasing the ratio of the special light image in an area that includescapillaries and a minute mucous membrane pattern in the mucous membranesurface layer in a high ratio. Moreover, the visibility of an area thatincludes capillaries and a minute mucous membrane pattern in the mucousmembrane surface layer in a low ratio can be improved by increasing theratio of the bright normal light image that contains a small amount ofnoise. This makes it possible to reduce the burden on the doctor whilepreventing a situation in which the lesion area is missed.

The edge extraction section 3421 shown in FIG. 12 extracts the edgeinformation about the corrected special light image by performing theedge detection filtering process including at least thedirection-dependent filtering process on the corrected special lightimage. For example, the edge extraction section 3421 extracts the edgeinformation about the corrected special light image by performing theLaplacian filtering process and the direction-dependent filteringprocess on the corrected special light image. This process correspondsto the lapImgG/AveG calculation process shown in FIG. 3.

The direction-dependent filtering process may be a filtering processthat smoothes the pixel values of the pixels positioned along the edgedirection. Specifically, the edge direction is determined based on theexpressions (8) to (15) from the eight directions shown in FIGS. 14A to14D.

The edge information can be extracted by performing the edge detectionfiltering process including the direction-dependent filtering process.For example, the edge image can be acquired by performing the Laplacianfiltering process, and noise can be reduced by performing thedirection-dependent filtering process. Therefore, an image having asharp edge can be acquired as the edge information about the correctedspecial light image.

The edge extraction section 3421 may extract the edge information aboutthe normal light image by performing the Laplacian filtering process andthe direction-dependent filtering process on the normal light image.This process corresponds to the lapImgY/AveY calculation process shownin FIG. 19.

For example, the edge image can be acquired by performing the Laplacianfiltering process, and noise can be reduced by performing thedirection-dependent filtering process. Therefore, an image having asharp edge can be acquired as the edge information about the normallight image.

The blending section 342 may include the attention area detectionsection 3425 and the blending ratio calculation section 3422, as shownin FIG. 13. The attention area detection section 3425 may detect theattention area within the corrected special light image, and theblending ratio calculation section 3422 may calculate the blending ratiobased on the detected attention area information. The blending section342 may perform the blending process based on the calculated blendingratio. For example, the attention area is detected by generating apseudo-color image from the signals G2 and B2 of the special lightimage, and determining the hue H.

The visibility of the lesion area can be improved as compared with thecase of using only normal light by improving the visibility of theattention area (e.g., lesion area) within the special light image, anddisplaying the pixels other than the attention area using the normallight image.

The blending ratio calculation section 3422 shown in FIG. 13 mayincrease the blending ratio of the corrected special light image whenthe attention area has been detected as compared with the case where theattention area has not been detected.

This makes it possible to improve the visibility of the lesion area orthe like as a result of increasing the ratio of the special light imagein the attention area. Moreover, the visibility of an area other thanthe attention area is improved since the ratio of the bright normallight image that contains a small amount of noise increases.

The blending section 342 may include a division section that divides thecorrected special light image into a plurality of areas. The blendingratio calculation section 3422 may calculate the blending ratio in eachof the plurality of areas.

This makes it possible to perform the process in units of areasincluding a plurality of pixels. Therefore, the amount of calculationscan be reduced, so that the processing speed can increased. Moreover,the blending ratio does not change to a large extent depending on thepixel as compared with the case where the corrected special light imageis not divided into a plurality of areas, and an artifact does not occureven when a pixel that includes capillaries and a minute mucous membranepattern in the mucous membrane surface layer is adjacent to a pixel thatdoes not include capillaries and a minute mucous membrane pattern in themucous membrane surface layer. This makes it possible to improve thevisibility of the lesion area, so that the burden on the doctor can bereduced while preventing a situation in which the lesion area is missed.

The term “specific wavelength band” may be a band that is narrower thanthe wavelength band of white light. Specifically, the normal light imageand the special light image may be in vivo images, and the specificwavelength band may be the wavelength band of a wavelength absorbed byhemoglobin in blood, for example. More specifically, the specificwavelength band may be 390 to 445 nm or 530 to 550 nm (see FIG. 7).

This makes it possible to observe the structure of a surface area oftissues and a blood vessel located in a deep area. A lesion area (e.g.,epidermoid cancer) that cannot be easily observed using normal light canbe displayed as a brown area or the like by inputting the resultingsignal to a given channel (R, G, or B), so that the lesion area can bereliably detected (i.e., a situation in which the lesion area is missedcan be prevented). A wavelength band of 390 to 445 nm or 530 to 550 nmis selected from the viewpoint of absorption by hemoglobin and theability to reach a surface area or a deep area of tissues.

As shown in FIGS. 5 and 11, the image processing device according tothis embodiment may include the normal light image acquisition section320, the special light image acquisition section 330, and the correctionsection 341. The normal light image acquisition section 320 may acquirean image that includes an object image and includes information withinthe wavelength band of white light, and the special light imageacquisition section 330 may acquire an image that includes an objectimage and includes information within a specific wavelength band. Thecorrection section 341 may perform the correction process on the speciallight image based on special light luminance information that is theluminance information about the special light image. The special lightluminance information corresponds to the luminance information nbiAveGshown in FIG. 3.

This makes it possible to correct the luminance of the special lightimage using the special light luminance information. Since the speciallight image is normally very dark, the visibility of the image can beimproved by correcting the luminance.

The correction section 341 shown in FIG. 11 may perform the correctionprocess on the special light image using the special light luminanceinformation and normal light luminance information that is the luminanceinformation about the normal light image. The normal light luminanceinformation corresponds to the luminance information wliAveG shown inFIG. 3.

This makes it possible to correct the luminance of the special lightimage after comparing the luminance of the normal light image with theluminance of the special light image.

The correction section 341 may cause the special light luminanceinformation to be equal to the normal light luminance information.

This makes it possible to increase the luminance of a very dark speciallight image to be equal to the luminance of the normal light image, sothat the visibility of the image can be improved.

The correction section 341 may perform the correction process on thespecial light image using the special light luminance information andluminance information obtained from a luminance sensor.

The luminance sensor (light control sensor) is implemented by aphotodiode or the like, and operates due to photoelectromotive force,photoconduction, or the like to detect the luminance.

This makes it possible to acquire the luminance information as sensorinformation, and correct the luminance based on the sensor information.

The correction section 341 may perform the correction process on thespecial light image based on the pixel value within the special lightimage and the pixel value within the normal light image.

This makes it possible to implement the correction process based on thepixel value. When 8 bits are assigned to R, G, and B of each pixel, thepixel value is a value of 0 to 255 corresponding to each of thecomponents R, G, and B.

The correction section 341 may calculate the special light luminanceinformation based on the pixel value within the special light image, andmay calculate the normal light luminance information based on the pixelvalue within the normal light image. The correction section 341 mayperform the correction process on the special light image based on thespecial light luminance information and the normal light luminanceinformation. The special light luminance information corresponds to theluminance information nbiAveG shown in FIG. 3, and the normal lightluminance information corresponds to the luminance information wliAveGshown in FIG. 3, as described above.

This makes it possible to calculate the luminance information based onthe pixel value. The luminance of the special light image can becorrected after comparing the luminance of the normal light image withthe luminance of the special light image.

The correction section 341 may calculate a value corresponding to theratio of the special light luminance information to the normal lightluminance information as a correction coefficient, and may multiply thepixel value within the special light image by the correctioncoefficient. For example, the process based on the expressions (3) and(4) may be employed.

This makes it possible to implement the correction process using theratio of the special light luminance information to the normal lightluminance information (or a value corresponding to the ratio of thespecial light luminance information to the normal light luminanceinformation) as the correction coefficient.

The normal light image may include first to Nth pixel-value components(e.g., components R, G, and B), and the special light image may includefirst to Mth pixel-value components (e.g., components G2 and B2). Thecorrection section 341 may calculate the special light luminanceinformation based on at least one of the first to Mth components. Thecorrection section 341 may calculate the normal light luminanceinformation based on the component of the normal light imagecorresponding to the component used to calculate the special lightluminance information. For example, the component of the normal lightimage corresponding to the component used to calculate the special lightluminance information may be at least one of the components G and B. Asshown in FIG. 3, when the signal G2 has been used to calculate thespecial light luminance information, the component G corresponds to thecomponent used to calculate the special light luminance information.

This makes it possible to calculate the special light luminanceinformation based on at least one of the pixel-value components of thespecial light image. Moreover, the normal light luminance informationcan be calculated based on the pixel-value component of the normal lightimage corresponding to the pixel-value component of the special lightimage used to calculate the special light luminance information. Whenthe component G2 has been used to calculate the special light luminanceinformation, the component G of the normal light image is used tocalculate the normal light luminance information. When the component B2has been used to calculate the special light luminance information, thecomponent B of the normal light image is used to calculate the normallight luminance information.

The correction section 341 may calculate the special light luminanceinformation based on the component G (component G2) of the special lightimage, and may calculate the normal light luminance information based onthe component G of the normal light image. The correction section 341may perform the correction process based on the special light luminanceinformation and the normal light luminance information. Specifically,the correction section 341 may calculate a value corresponding to theratio of the special light luminance information to the normal lightluminance information as a correction coefficient, and may multiply thecomponents G and B of the special light image by the correctioncoefficient. For example, the process based on the expressions (3) to(5) may be employed.

This makes it possible to increase the luminance of the special lightimage to be almost equal to the luminance of the normal light imageusing the signal G with the highest relative luminous efficiency. Thismakes it possible to improve the visibility of the lesion area, so thatthe burden on the doctor can be reduced while preventing a situation inwhich the lesion area is missed.

The correction section 341 may calculate first special light luminanceinformation based on the component G (component G2) of the special lightimage, and may calculate first normal light luminance information basedon the component G of the normal light image. The correction section 341may calculate second special light luminance information based on thecomponent B (component B2) of the special light image, and may calculatesecond normal light luminance information based on the component B ofthe normal light image. The correction section 341 may perform thecorrection process based on the first special light luminanceinformation, the first normal light luminance information, the secondspecial light luminance information, and the second normal lightluminance information.

Specifically, the correction section 341 may calculate a valuecorresponding to the ratio of the first special light luminanceinformation to the first normal light luminance information as a firstcorrection coefficient, and may multiply the component G of the speciallight image by the first correction coefficient. The correction section341 may calculate a value corresponding to the ratio of the secondspecial light luminance information to the second normal light luminanceinformation as a second correction coefficient, and may multiply thecomponent B of the special light image by the second correctioncoefficient. For example, the process based on the expressions (3), (4),(6), and (7) may be employed.

This makes it possible to optimally correct the luminance individuallyusing the signals G and B. Therefore, the visibility of the lesion areacan be improved, so that the burden on the doctor can be reduced whilepreventing a situation in which the lesion area is missed.

The blending section 342 may blend the component G of the normal lightimage and the component G of the corrected special light image, andblend the component B of the normal light image and the component B ofthe corrected special light image. This corresponds to C1 in FIG. 3 andD1 in FIG. 4.

This makes it possible to blend the special light image having acorrected luminance with the normal light image. Since the luminance ofthe special light image that is normally very dark has been corrected,the normal light image does not become dark even if the blending processis performed, so that the visibility of the lesion area can be improved.

The normal light image acquisition section 320 may acquire the normallight image based on light from a light source having the wavelengthband of white light. The special light image acquisition section 330 mayacquire the special light image based on light from a light sourcehaving the specific wavelength band. In this case, a white light sourceand a special light source are provided in the light source section 100instead of the white light source 110.

This makes it possible to acquire the normal light image and the speciallight image based on the configuration of the light source. Since it isunnecessary to apply a filter or the like, the configuration of theinsertion section 200 and the like can be simplified.

The normal light image acquisition section 320 may acquire the normallight image based on light from the light source using a filter thatallows light having the wavelength band of white light to pass through.The special light image acquisition section 330 may acquire the speciallight image based on light from the light source using a filter thatallows light having the specific wavelength band to pass through.Specifically, the configuration shown in FIG. 5 may be employed. Theimaging element 250 includes the color filter shown in FIG. 6, and theimaging element 260 includes the color filter shown in FIG. 7.

This makes it possible to acquire the normal light image and the speciallight image based on the configuration of the filter. In this case, asingle light source that covers the wavelength band of white light andthe wavelength band of special light can be used. Therefore, theconfiguration of the light source section can be simplified.

This embodiment also relates to an image processing method includingacquiring a normal light image that includes an object image andincludes information within the wavelength band of white light,acquiring a special light image that includes an object image andincludes information within a specific wavelength band, performing acorrection process on the special light image, and performing at leastone of a first blending process that blends the component G of thenormal light image and the component G of the corrected special lightimage, and a second blending process that blends the component B of thenormal light image and the component B of the corrected special lightimage as a blending process that blends the normal light image and thecorrected special light image.

An image in which the visibility of capillaries and a minute mucousmembrane pattern in the mucous membrane surface layer is higher thanthat of the normal light image can be obtained by blending the speciallight image corrected by the correction section 341 with the normallight image. Moreover, since the components G and the components B arerespectively blended, the visibility of the blood vessels and the likecan be improved with a natural color.

This embodiment also relates to a program that causes a computer tofunction as the normal image acquisition section 320, the special lightimage acquisition section 330, the correction section 341, and theblending section 342. The normal light image acquisition section 320acquires an image that includes an object image and includes informationwithin the wavelength band of white light, and the special light imageacquisition section 330 acquires an image that includes an object imageand includes information within a specific wavelength band. Thecorrection section 341 performs the correction process on the speciallight image, and the blending section 342 performs at least one of ablending process that blends the component G of the normal light imageand the component G of the corrected special light image, and a blendingprocess that blends the component B of the normal light image and thecomponent B of the corrected special light image.

This makes it possible to store image data (e.g., capsule endoscope),and process the stored image data by software using a computer system(e.g., PC).

This embodiment also relates to a program that causes a computer tofunction as the normal image acquisition section 320, the special lightimage acquisition section 330, and the correction section 341. Thenormal light image acquisition section 320 acquires an image thatincludes an object image and includes information within the wavelengthband of white light, and the special light image acquisition section 330acquires an image that includes an object image and includes informationwithin a specific wavelength band. The correction section 341 performsthe correction process on the special light image based on the speciallight luminance information.

This makes it possible to store image data (e.g., capsule endoscope),and process the stored image data by software using a computer system(e.g., PC).

This embodiment also relates to a computer program product that stores aprogram code that implements each section (normal light imageacquisition section, special light image acquisition section, correctionsection, and blending section) according to this embodiment.

The program code implements the normal light image acquisition sectionthat acquires a normal light image that includes an object image andincludes information within the wavelength band of white light, thespecial light image acquisition section that acquires a special lightimage that includes an object image and includes information within aspecific wavelength band, the correction section that performs acorrection process on the special light image, and the blending sectionthat performs a blending process that blends the component G of thenormal light image and the component G of the corrected special lightimage, and a blending process that blends the component B of the normallight image and the component B of the corrected special light image asa blending process that blends the normal light image and the correctedspecial light image that is the special light image corrected by thecorrection section.

The term “computer program product” used herein refers to an informationstorage medium, a device, an instrument, a system, or the like thatstores a program code, such as an information storage medium (e.g.,optical disk medium (e.g., DVD), hard disk medium, and memory medium)that stores a program code, a computer that stores a program code, or anInternet system (e.g., a system including a server and a clientterminal), for example. In this case, each element and each processaccording to this embodiment are implemented by corresponding modules,and a program code that includes these modules is recorded in thecomputer program product.

3. Second Embodiment

A method according to this embodiment is described below with referenceto FIGS. 19 and 20.

FIG. 19 is a view illustrative of a component G blending process. Thespecial light image includes components G2 and B2 (when the speciallight image is an NBI image), and the component G2 is extracted. Thenormal light image includes components R, G, and B, and the component Gthat corresponds to the component G2 is extracted.

The ratio ratioG is calculated by the expression (3) from the ratio ofthe luminance information wliAveG about the component G of the normallight image to the luminance information nbiAveG about the component G2of the special light image. The luminance of the component G2 of thespecial light image is corrected by the expression (4) using the ratioratioG to acquire the corrected special light image (component G).

A noise reduction process is performed on the corrected special lightimage using an edge information image edgeY of the normal light image toacquire a noise reduction image wliNrImg. The noise reduction process isalso performed on the corrected special light image using an edgeinformation image edgeNbi of the normal light image to acquire a noisereduction image nbiNrImg.

The noise reduction image wliNrImg and the noise reduction imagenbiNrImg are blended using the expression (30) (described later) toacquire a noise reduction special light image (component G).

The noise reduction special light image (component G) and the normallight image (component G) are blended using the blending ratio blendG toacquire an output image (component G). Note that the process accordingto this embodiment aims at acquiring the output image. The edgeinformation image edgeNbi, the edge information image edgeY, and theblending ratio blendG are calculated the following process.

When calculating the edge information image edgeNbi, it is necessary tocalculate the edge information image edgeG of the component G of thespecial light image and the edge information image edgeB of thecomponent B of the special light image. The edge information image edgeGand the edge information image edgeB can be calculated by performing theLaplacian filtering process, the direction-dependent filtering process,and the coring process in the same manner as in the first embodiment.The edge information image edgeNbi is calculated by the expression (27)or (28) (described later) using the edge information images edgeG andedgeB thus calculated. The blending ratio blendG can be acquired bymultiplying the edge information image edgeG by the coefficient K1.

When calculating the edge information image edgeY, a luminance image isacquired by the expression (26) (described later) using the componentsR, G, and B of the normal light image. The edge information image edgeYis obtained by subjecting the luminance image to the Laplacian filteringprocess, the direction-dependent filtering process, and the coringprocess in the same manner as in the case of the special light image.

FIG. 20 schematically shows a component B blending process.Specifically, the corrected special light image (B component) isacquired from the component B2 of the special light image, and subjectedto the noise reduction process using the edge information image edgeNbiand the edge information image edgeY to acquire a noise reductionspecial light image (B component). The noise reduction special lightimage (component B) and the normal light image (component B) are blendedusing the blending ratio blendB to acquire an output image (componentB).

The component R of the normal light image is directly used in the samemanner as in the first embodiment.

A color image is generated based on the components R, B, and G of theoutput image thus obtained. Since the blood vessels and the like areenhanced in a natural color in the generated image, and noise has beenreduced, the burden on the doctor during diagnosis can be reduced, forexample.

The system configuration and the above process are described in detailbelow. Note that detailed description of the configuration and theprocess similar to those of the first embodiment is omitted.

The configuration according to this embodiment is the same as theconfiguration according to the first embodiment except for the blendingsection 342.

The blending section 342 blends the corrected special light image withthe normal light image under control of the control section 360. Aspecific configuration of the blending section 342 is described later.FIG. 21 is a block diagram illustrative of an example of theconfiguration of the blending section 342 according to the secondembodiment. As shown in FIG. 21, the blending section 342 according tothe second embodiment further includes a noise reduction section 3424.

The correction section 341 is connected to the edge extraction section3421 and the noise reduction section 3424. The edge extraction section3421 is connected to the blending ratio calculation section 3422 and thenoise reduction section 3424. The noise reduction section 3424 isconnected to the blended image generation section 3423. The controlsection 360 is bidirectionally connected to the edge extraction section3421, the blending ratio calculation section 3422, the blended imagegeneration section 3423, and the noise reduction section 3424, andcontrols the edge extraction section 3421, the blending ratiocalculation section 3422, the blended image generation section 3423, andthe noise reduction section 3424.

The edge extraction section 3421 extracts the edge information about thecorrected special light image and the edge information about the normallight image. The edge extraction section 3421 performs the directiondetermination edge extraction process on the luminance signal value Y ofthe normal light image, the G signal value of the special light image,and the B signal value of the special light image. Specifically, theedge extraction section 3421 calculates the luminance signal value Y ofthe normal light image by the following expression (26) using the R, G,and B signal values.Y(x, y)=0.213×r(x, y)+0.715×g(x, y)+0.072×b(x, y)  (26)

The edge extraction section 3421 then performs the process performed onthe signals G and B of the special light image (refer to the firstembodiment) on the luminance signal value Y of the normal light image.The edge image edgeY of the luminance signal value Y of the normal lightimage is output to the noise reduction section 3424. The edge imageedgeG of the G signal value of the special light image and the edgeimage edgeB of the B signal value of the special light image generatedby a similar process are output to the blending ratio calculationsection 3422 and the noise reduction section 3424.

The noise reduction section 3424 performs a noise reduction process onthe corrected special light image under control of the control section360 based on the edge information edgeY about the normal light image orthe edge images edgeG and edgeB of the corrected special light image.Specifically, the noise reduction section 3424 blends the edge imageedgeG of the G signal value of the special light image and the edgeimage edgeB of the B signal value of the special light image using thefollowing expression to generate the edge image edgeNbi of the speciallight image.edgeNbi(x, y)=MAX(edgeG(x, y),edgeB(x, y))  (27)

MAX(a, b) is a function that outputs the larger of a and b. Note thatthe edge images may be blended as follows.edgeNbi(x, y)=K3×edgeG(x, y)+(1−K3)×edgeB(x, y)  (28)

Note that K3 is a constant term in the range of 0 to 1. A value input inadvance from the outside is used as K3.

The noise reduction section 3424 then performs a first noise reductionprocess on the corrected special light image. Specifically, the noisereduction section 3424 performs a weighted summation process whileadding a large weight to a proximate pixel having a value close to thatof the attention pixel, and adding a small weight to a proximate pixelhaving a value that is not close to that of the attention pixel. Thenoise reduction section 3424 generates a first noise reduction imagenbiNrImg by performing a filtering process that implements a weightedsummation process while adding a large weight to a proximate pixelhaving a value close to that of the attention pixel in the edge imageedgeNbi of the special light image, and adding a small weight to aproximate pixel having a value that is not close to that of theattention pixel in the edge image edgeNbi of the special light image.

The noise reduction section 3424 then performs a second noise reductionprocess on the corrected special light image that has not been subjectedto the noise reduction process. Specifically, the noise reductionsection 3424 generates a second noise reduction image wliNrImg byperforming a Gaussian filtering process that decreases the value sigmaof the filter when the signal value within the edge image edgeY of thenormal light image corresponding to the attention pixel is large, andincreases the value sigma of the filter when the signal value within theedge image edgeY is small.

The first noise reduction process is complex as compared with the secondnoise reduction process since the first noise reduction process is aprocess based on the edge information about the special light image.Since the special light image is very dark and contains a large amountof noise, noise is carefully reduced by utilizing a complex process sothat the process is not affected by noise. Since the first noisereduction process is based on the normal light image that is bright andcontains a small amount of noise, only the Gaussian filtering process isused.

The first noise reduction image nbiNrImg and the second noise reductionimage wliNrImg are then blended.K5(x, y)=K4×edgeNbi(x, y)  (29)nrImg(x, y)=K5(x, y)×nbiNrImg(x, y)+(1−K5(x, y))×wliNrImg(x, y)  (30)

Note that K4 is a constant term. K5 is a value in the range of 0 to 1.The image nrImg obtained by the noise reduction process is output to theblended image generation section 3423.

The reasons why the first noise reduction image nbiNrImg and the secondnoise reduction image wliNrImg are used to calculate the image nrImg,and the edge information image edgeNbi is used when calculating theblending ratio K5 are describe below. In the special light image, astructure (e.g., blood vessel) is enhanced as compared with the normallight image. Therefore, it is preferable that the edge information aboutthe special light image be preferentially reflected in the output imageas compared with the edge information about the normal light image.Therefore, the edge information image edgeNbi is used when calculatingthe blending ratio K5. However, the special light image is very dark,and contains a large amount of noise. When using only the edgeinformation about the special light image, the process may be adverselyaffected by noise. The effects of noise are reduced by utilizing theedge information about the normal light image that is bright andcontains a small amount of noise.

In this embodiment, each section of the image processing section 300 isimplemented by hardware. Note that a CPU may perform the process of eachsection on an image acquired in advance in the same manner as in thefirst embodiment. Specifically, the process of each section may beimplemented by software by causing the CPU to execute a program.Alternatively, part of the process of each section may be implemented bymeans of software.

In this case, the process is the same as that of the first embodimentexcept for the blending process (S16) shown in FIG. 17. The details ofthe blending process are described below with reference to FIG. 22.

Specifically, the edge information about the corrected special lightimage and the edge information about the normal light image areextracted using the expressions (8) to (20) (S31). The noise reductionprocess is then performed on the corrected special light image using theexpressions (29) and (30) (S32). The blending ratio is calculated basedon the extracted edge information using the expressions (21) and (22)(S33). A blended image is then generated by blending the signals B and Gof the normal light image with the signals B and G of the special lightimage subjected to the noise reduction process using the expressions(23) to (25) (S34).

The above process makes it possible to display the normal light imageand the special light image as a single image. Therefore, it is possibleto provide an endoscope system that prevents a situation in which alesion area is missed while reducing the burden on the doctor.

According to this embodiment, the noise reduction section 3424 shown inFIG. 21 performs the noise reduction process on the corrected speciallight image based on the edge information about the normal light imageor the edge information about the corrected special light image. Theblending section 342 blends the special light image subjected to thenoise reduction process with the normal light image. This corresponds toE1 in FIG. 19 and F1 in FIG. 20.

This makes it possible to reduce the noise level of the special lightimage that has been increased by the correction process. Therefore, aspecial light image having a low noise level can be blended with thenormal light image. The visibility of the lesion area can be improved bygenerating a blended image having a low noise level as compared with thecase where the noise reduction process is not performed, so that theburden on the doctor can be reduced while preventing a situation inwhich the lesion area is missed.

The noise reduction section 3424 controls the degree of the noisereduction process based on the edge information about the normal lightimage. The edge information about the normal light image corresponds tothe edge information image edgeY shown in FIGS. 19 and 20. The noisereduction image wliNrImg is acquired by performing the noise reductionprocess on the corrected special light image using the edge informationimage edgeY.

The degree of the noise reduction process refers to the value sigma ofthe Gaussian filtering process, for example.

Therefore, noise can be reduced without impairing the blood vesselinformation included in the normal light image. The visibility of thelesion area can be improved by generating a blended image having a lownoise level, so that the burden on the doctor can be reduced whilepreventing a situation in which the lesion area is missed.

The noise reduction section 3424 may perform an edge directiondetermination noise reduction process based on the edge informationabout the corrected special light image. The edge information about thecorrected special light image corresponds to the edge information imageedgeNbi shown in FIGS. 19 and 20. The noise reduction image nbiNrImg isacquired by performing the noise reduction process on the correctedspecial light image using the edge information image edgeNbi.

Therefore, noise can be reduced without impairing the image ofcapillaries and a minute mucous membrane pattern in the mucous membranesurface layer even if the special light image is very dark. Thevisibility of the lesion area can be improved by generating a blendedimage having a low noise level, so that the burden on the doctor can bereduced while preventing a situation in which the lesion area is missed.

The noise reduction section 3424 may perform a weighted summationprocess as the edge direction determination noise reduction processwhile adding a large weight to a pixel having a value close to that ofthe processing target pixel, and adding a small weight to a pixel havinga value that is not close to that of the processing target pixel. Thiscorresponds to the nbiNrImg calculation process shown in FIGS. 19 and20.

This makes it possible to perform the noise reduction process based onthe edge direction without impairing the image of capillaries and aminute mucous membrane pattern in the mucous membrane surface layer evenif the special light image is very dark.

The first and second embodiments according to the invention and themodifications thereof have been described above. Note that the inventionis not limited to the first and second embodiments and the modificationsthereof. Various modifications and variations may be made withoutdeparting from the scope of the invention. A plurality of elements ofeach of the first and second embodiments and the modifications thereofmay be appropriately combined. For example, some elements may be omittedfrom the elements of the first and second embodiments and themodifications thereof. The elements described in connection with theabove embodiments and the modifications thereof may be appropriatelycombined. Specifically, various modifications and applications arepossible without materially departing from the novel teachings andadvantages of the invention.

Any term (e.g., narrow-band light) cited with a different term having abroader meaning or the same meaning (e.g., special light) at least oncein the specification and the drawings can be replaced by the differentterm in any place in the specification and the drawings.

Although only some embodiments of the invention have been described indetail above, those skilled in the art would readily appreciate thatmany modifications are possible in the embodiments without materiallydeparting from the novel teachings and advantages of the invention.Accordingly, such modifications are intended to be included within thescope of the invention.

What is claimed is:
 1. An image processing device comprising: aprocessor; and a memory storing computer readable instructions that,when executed by the processor, cause the processor to: acquire a normallight image that includes information within a wavelength band of whitelight; acquire a special light image that includes information within aspecific wavelength band; perform a correction process on the speciallight image and generate a corrected special light image based on thespecial light image that is corrected; detect an attention area withinthe corrected special light image; and perform a blended imagegeneration process of generating a blended image based on the normallight image and the corrected special light image, wherein the blendedimage generation process comprises: calculating a contribution of thecorrected special light image to the blended image, wherein thecontribution of the corrected special light image to the blended imageis increased when the attention area has been detected within thecorrected special light image as compared with the case where theattention area has not been detected within the corrected special lightimage; and performing at least one of a first blending process and asecond blending process based on the contribution of the correctedspecial light image calculated, wherein the first blending processcomprises blending a component G of the normal light image and acomponent G of the corrected special light image, and wherein the secondblending process comprises blending a component B of the normal lightimage and a component B of the corrected special light image.
 2. Theimage processing device as defined in claim 1, wherein the specificwavelength band is narrower than the wavelength band of the white light.3. An image processing method comprising: acquiring a normal light imagethat includes information within a wavelength band of white light;acquiring a special light image that includes information within aspecific wavelength band; performing a correction process on the speciallight image and generating a corrected special light image based on thespecial light image that is corrected; detecting an attention areawithin the corrected special light image; and performing a blended imagegeneration process of generating a blended image based on the normallight image and the corrected special light image, wherein the blendedimage generation process comprises: calculating a contribution of thecorrected special light image to blended image, wherein the contributionof the corrected special light image is increased when the attentionarea has been detected within the corrected special light image ascompared with the case where the attention area has not been detectedwithin the corrected special light image; and performing at least one ofa first blending process and a second blending process based on thecontribution of the corrected special light image calculated, whereinthe first blending process comprises blending a component G of thenormal light image and a component G of the corrected special lightimage, and wherein the second blending process comprises blending acomponent B of the normal light image and a component B of the correctedspecial light image.
 4. A non-transitory computer readable storagedevice with executable program stored thereon, wherein the programinstructs a microprocessor to perform functions comprising: acquiring anormal light image that includes information within a wavelength band ofwhite light; acquiring a special light image that includes informationwithin a specific wavelength band; performing a correction process onthe special light image and generating a corrected special light imagebased on the special light image that is corrected; detecting anattention area within the corrected special light image; and performinga blended image generation process of generating a blended image basedon the normal light image and the corrected special light image, whereinthe blended image generation process comprises: calculating acontribution of the corrected special light image to the blended image,wherein the contribution of the corrected special light image to theblended image is increased when the attention area has been detectedwithin the corrected special light image as compared with the case wherethe attention area has not been detected within the corrected speciallight image; and performing at least one of a first blending process anda second blending process based on the contribution of the correctedspecial light image calculated, wherein the first blending processcomprises blending a component G of the normal light image and acomponent G of the corrected special light image, and wherein the secondblending process comprises blending a component B of the normal lightimage and a component B of the corrected special light image.